Skip to content

Leave an operation or API alone when the API asks it to wait - #424

Merged
ssegura merged 2 commits into
masterfrom
feat/m13-2a-wait-when-told-to-wait
Oct 29, 2026
Merged

ssegura merged 2 commits into
masterfrom
feat/m13-2a-wait-when-told-to-wait

Conversation

@ssegura

@ssegura ssegura commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Related: None. The second of the safeguards planned for 2.4.0.

What you can do now that you could not before

An API that asks RESTest to slow down is now obeyed. When an operation answers 429 Too Many Requests, or 503 with a Retry-After header, RESTest sends that operation nothing for as long as
the API asks, while it goes on testing the others. When a second operation is told to wait while
another still waits, the limit is plainly the whole API's, and nothing at all is sent until it is
over.

Before, a 429 was an ordinary answer, and a quick one, so the engine took a stream of them as room
to send more. Against an API limited to 20 requests a second, RESTest 2.3.0 sent 1,193,708 requests
inside the quiet the API had asked for, in 20 seconds. This branch sends 277.

Try it yourself

From a clean clone, with only Java and git. A one-file Java server stands in for an API that lets 20
requests a second through, answers 429 with Retry-After: 1 to the rest, and counts the requests
that arrive while the quiet it asked for lasts:

git fetch && git switch feat/m13-2a-wait-when-told-to-wait
./mvnw -q install -DskipTests
cat > Limit.java <<'EOF'
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.util.concurrent.Executors;

/**
 * An API that lets 20 requests a second through and answers 429, asking for a second's quiet, to
 * the rest. When it is stopped it says how many requests arrived while that quiet lasted.
 */
public class Limit {
    private static long second, inThisSecond, quietUntil, answered, refused, duringTheQuiet;

    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress(8429), 0);
        server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
        server.createContext("/", exchange -> {
            long now = System.currentTimeMillis();
            boolean let;
            synchronized (Limit.class) {
                if (now < quietUntil) {
                    duringTheQuiet++;
                    let = false;
                } else {
                    if (now / 1000 != second) {
                        second = now / 1000;
                        inThisSecond = 0;
                    }
                    let = ++inThisSecond <= 20;
                    if (!let) {
                        quietUntil = now + 1000;
                    }
                }
                if (let) answered++; else refused++;
            }
            if (let) {
                byte[] body = "[]".getBytes();
                exchange.getResponseHeaders().add("Content-Type", "application/json");
                exchange.sendResponseHeaders(200, body.length);
                exchange.getResponseBody().write(body);
            } else {
                exchange.getResponseHeaders().add("Retry-After", "1");
                exchange.sendResponseHeaders(429, -1);
            }
            exchange.close();
        });
        Runtime.getRuntime().addShutdownHook(new Thread(() -> System.out.println(answered
                + " answered, " + refused + " refused, " + duringTheQuiet
                + " of them while the quiet it asked for lasted")));
        server.start();
        System.out.println("letting 20 requests a second through at http://localhost:8429");
    }
}
EOF
java Limit.java &
./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://localhost:8429 --budget 20s; echo "exit $?"
grep -o '"toldToWait":{[^]]*]}' restest-out/report.json
kill %1

Below is the real output, from a fresh clone of this branch on 29 October 2026. The run prints its
findings first; only its summary is shown here. Two things are new. The first line ends with
(96% waiting as the API asked). The line starting told to wait: is new too.

692 requests to 4 operations in 20.0s, 98% of it idle (96% waiting as the API asked)
  opening lap: 4 requests in 182ms, 4 of 4 operations answered 2xx
  400 2xx, 289 4xx, 3 no reply
  27 of them were pushing at the API with values nobody sensible would send, which accounts for some of the 289 refusals above
  123 of them changed one thing in a request the API had accepted
  20 of them were steps of 13 series about a thing the run created: createTwice 14, safeGet 6; 7 creation(s) meant to begin one were not accepted
  told to wait: the whole API 20 times, 19.2s in all; listShelters 20 times, 19.2s in all, 1 turn skipped; listPets 20 times, 19.2s in all, 1 turn skipped; addPet 20 times, 19.2s in all, 4 turns skipped; and 1 more; the run's report names every one
68 errors, on 2 operations:
  68 x Invalid data accepted  (F206)
        make the API refuse what its document forbids; where the document forbids more than the API should, loosen the document instead
583 warnings, on 4 operations, which do not fail the run (checks.failOnWarnings):
  204 x Reply off schema  (F200)
        the API and its document disagree about the reply's body: compare the places listed with the document, and fix whichever is wrong
  379 x Undeclared status code  (F920)
        declare the status code for the operation in the document, or stop the API answering it
6 test case(s) could not be built and were skipped
report written to restest-out/report.json (81.2 KiB)
page written to restest-out/report.html (162.2 KiB)
the run itself was not kept; pass --store to keep every request and reply
exit 1
"toldToWait":{"idle":"PT19.244670338S","wholeApi":{"times":20,"waited":"PT19.234234S"},"operations":[{"operation":"listShelters","times":20,"stated":20,"waited":"PT19.225616S","turnsSkipped":1},{"operation":"listPets","times":20,"stated":20,"waited":"PT19.225329S","turnsSkipped":1},{"operation":"addPet","times":20,"stated":20,"waited":"PT19.229053S","turnsSkipped":4},{"operation":"getPet","times":20,"stated":20,"waited":"PT19.227509S","turnsSkipped":0}]}
400 answered, 289 refused, 269 of them while the quiet it asked for lasted

The last line is the stand-in's. Of the 289 requests it refused, 269 arrived while its quiet lasted.
They are the requests already on their way when each of its 20 quiet seconds began.

The exit code is 1 because pet-shelter accepts data its document forbids (F206), as it does on
master; waiting changes no exit code. The 3 no reply are requests the engine still held, for
quiet, when the run's time ran out. They were never sent.

With the switch off, the run sends as it did before, so this run is only 5 seconds long:

java Limit.java &
./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://localhost:8429 --budget 5s --set schedule.waitWhenToldTo=false; echo "exit $?"
kill %1
241366 requests to 4 operations in 5.0s, 8% of it idle
  100 2xx, 241266 4xx
exit 1
100 answered, 241266 refused, 241261 of them while the quiet it asked for lasted

(That run prints its findings too; only these lines are shown. Its report.json has "toldToWait":null.)

How it works

An API that is asked too much answers 429, "too many requests", or 503, "not now". It may say in
a Retry-After header when to ask again: a number of seconds, or a date. What it does not say is
whether the limit covers one operation or the whole API. HTTP allows both, and the APIs of the
dataset RESTest is measured on show both kinds:

  • cwa-verification's createTeleTan spends its quota of about 1,000 creations an hour within 3
    seconds. Its other four operations keep answering.
  • genome-nexus's fetchGeneXrefsGET passes on a third party's limit of 15 requests a second.

So RESTest leaves alone the operation that was told to wait, and skips its turns. It waits for as long
as the header says, or, when the header says nothing, for 1 second, then 2, 4 and so on up to a
minute. The wait goes back to 1 second as soon as the operation gives any answer that does not ask it
to wait.

createTeleTan -> 429          createTeleTan waits 1 s, 2 s, 4 s ... up to 60 s
                              the other operations go on as before

getA -> 429, then getB -> 429 while getA still waits
                              the whole API waits: nothing is sent, the engine holds what
                              it already had, then starts again from one request at a time

When a second operation is told to wait while another still waits, the limit is plainly the whole
API's. Then nothing at all is sent until the wait is over: neither the requests the run had not yet
handed to the engine (the part that sends) nor those already handed over. The engine then starts
again from one request at a time and climbs as it always does.

The request that drew the 429 is not sent again: every turn builds a new request anyway. A 503
without Retry-After is a server error like any other, and makes nothing wait. report.json says,
under toldToWait, which operations were told to wait, how often, for how long, and how many of
their turns were skipped. The summary says how much of the time the run sent nothing because the
API asked, so that this is never read as RESTest being slow. One switch, schedule.waitWhenToldTo,
on by default, turns it all off. The decision, the measurements and the alternatives are in
ADR-0042.

What changed in the code

  • restest-cli
    • RetryAfter (new) reads the header: seconds, or a date in the three forms HTTP says a reader
      must accept. A date is measured from the reply's own Date header when it has one.

      • Every wait is kept between now and the run's deadline, so no value can make it fail.
      • A wrong day name is ignored.
    • ToldToWait (new) keeps the waits, shaped like 13.1's RefusedOpeningLap.

      • RunLoop tells it the moment each request settles, from the engine's threads, inside the
        region that gives the request's slot back.
      • The loop asks it, before building each request, whether the operation may be sent and whether
        the whole API waits.
      • A refusal of a request sent before the current wait began starts nothing new. A
        stated Retry-After only moves the end of a wait still going later.
      • A volatile "latest end" means that once every wait is over, nothing takes the lock.
    • RunLoop:

      • skips a waiting operation's turn;
      • sits a whole-API wait out in looks of 50 ms, ending early at the deadline or at a stop from
        outside;
      • sits out a round in which every turn was skipped, until the soonest wait ends, or for 50 ms
        when a request of that round could not be built;
      • when it stops sending, has the engine drop what it still holds.

      Outcome gains waited.

    • RunCommand builds ToldToWait, puts its summary on RunFinished, and prints the line under
      the seed while waiting is on.

  • restest-exec
    • ConcurrencyLimiter.holdFor gives no slot until the hold ends and drops the limit to the floor;
      waiting out a hold earns no growth.
    • dropWhatIsHeld lets held requests go away unsent.
    • close() lets every request still waiting for a slot go away unsent.
    • OkHttpEngine passes these on. A request accepted just before the engine closes now comes back
      as never sent, instead of throwing on its own thread.
  • restest-core
    • HttpEngine gains holdFor(Duration) and dropWhatIsHeld(), abstract so that no wrapper can
      forget them; CredentialedEngine passes them on.
    • Interaction.responseHeaderValues(name) reads a reply's headers even when its body broke off.
    • Waited (new) rides on RunEvent.RunFinished as 13.1's Unauthorized does.
    • ScheduleSettings gains waitWhenToldTo, firstGuessedWait (1s) and longestGuessedWait
      (1m).
  • restest-gen: Scheduler.notSent hands a skipped step of a series back as one that could not
    be built, so the series goes on without it or ends.
  • restest-report
    • JsonReport writes toldToWait.
    • ConsoleReport adds the share in brackets and the told to wait: line.
    • The HTML page is unchanged: a section for the waits is a change of layout, which needs a mock-up
      first.
  • Docs.
    • New: ADR-0042.
    • Amended: ADR-0009 (idle time the API asked for), ADR-0016 (the 503 and 429 still reported, and
      fewer) and ADR-0025 (the settings).
    • docs/known-limitations.md, faults.md, report.md, settings.md and switches.md, the last
      including "Getting the seed back", now four things. Manual chapters 5 and 10; chapter 10 shows a
      real run against the stand-in.
  • Left for later, as possible extensions in no version yet:
    • stopping with 5 when the whole API keeps asking to wait, or asks past the budget;
    • reading the IETF draft's RateLimit and RateLimit-Policy fields;
    • a section on the page.

Evidence

  • Tests: ./mvnw verify is green at the tip of this branch: 4,887 tests, 0 failures, BUILD
    SUCCESS. The new ones:

    • RetryAfterTest: each form of the header, a two-digit year, a wrong day name, the reply's Date,
      values too large for any clock, and what cannot be read.
    • ToldToWaitTest: guessing and doubling, what brings the guess back, stale refusals, the whole
      API's own wait, hostile headers, the switch, the summary, and a race between the thread that
      answers and the thread that asks.
    • RunLoopTest, 9 new cases:
      • an operation skipped while the others go on;
      • the whole API waiting;
      • a wait ending at the deadline and holding the engine for good;
      • a stop from outside during a wait;
      • a round with everything waiting;
      • a wait during the opening lap;
      • a series going on past a skipped step;
      • a broken hold losing nothing;
      • the switch off.
    • ToldToWaitCommandTest: 5 whole runs against WireMock, read from its record of every request:
      • nothing to a waiting operation inside its wait, while the others go on;
      • nothing at all inside a wait of the whole API;
      • nothing after the run's end when the quiet lasts to it;
      • the switch off;
      • the line under the seed.
    • ConcurrencyLimiterTest and OkHttpEngineTest: hold, extend, drop and close.
    • The rest are in SchedulerTest, InteractionTest, CredentialedEngineTest, SettingsTest,
      JsonReportTest and ConsoleReportTest.

    CI: on this pull request.

  • Architecture tests: DocumentedSwitchesTest now also holds the file for getting the seed
    back to the waiting switch; DocumentedReportTest and DocumentedSettingsTest hold the new keys and
    settings to their pages. None weakened.

  • Mutation score: no oracle or constraint touched.

  • Per-request overhead: none to be seen. blogapi without its key, 20 s, sends 146,147
    requests on master and 149,242 on this branch.

  • Smoke run: CI runs it on this pull request.

  • Idle time: 1.9% on master, 1.5% on this branch, for the same blogapi run. Nothing waited
    there.

  • Artefact. Every stored report of the Web Fuzzing Dataset's 52 APIs, run locally, was
    scanned first:

    • two APIs ever answer 429, each for one operation only, with no Retry-After;
    • three answer 503, none with Retry-After.

    Replayed apart from RESTest's code over runs stored with 2.3.0, 60 s each, with a first wait of
    1 s doubling to 60 s:

    Every request to the API waits Only the operation told to wait waits
    cwa-verification 57 s of 60 waiting; 2xx replies from 9,275 to 1,002 no 2xx lost
    genome-nexus 27 s waiting no 2xx lost

    Then built, and run against master, same seed:

    Run master This branch
    cwa-verification, 60 s: replies of 429 34,699 62, in 12 waits of createTeleTan; 57,728 of its turns skipped
    cwa-verification, 60 s: 2xx replies, operations answering 2xx 8,756, 3 of 5 9,503, 3 of 5
    cwa-verification, 60 s, with the switch off — 34,931 replies of 429, 8,778 2xx
    genome-nexus, 60 s: replies of 429 5 and 1 (two runs) 6; fetchGeneXrefsGET told to wait 4 times
    genome-nexus, 60 s: 2xx replies, operations answering 2xx 2,735 and 1,766; 17 and 16 of 23 2,214, 17 of 23
    The stand-in above, 20 s: requests inside the quiet it asked for 1,193,708 277

    genome-nexus calls a service of its own over the internet, and its runs vary that much with nothing
    changed. jasper's 293 replies of 503 and commafeed's 421, from runs of 20 s with every request
    kept, carry no Retry-After, and nothing waited there.

Decisions taken

  • The operation waits, not the whole API. The plan's row said every request waits at the first
    429. The data above showed that this would leave cwa-verification waiting 57 of its 60 seconds,
    for a quota of one operation. The maintainer chose the operation, with the whole API waiting once a
    second operation is told while another waits.
  • Only this part of the row now. Stopping with 5 when the whole API keeps asking to wait, and
    the RateLimit fields, are left as possible extensions (maintainer). None of the dataset's APIs
    makes the whole API wait or sends those fields.
  • A 503 waits only with Retry-After. This was asked and confirmed by the maintainer. If every
    503 waited, jasper's four broken operations, which answer 503 together, would make the whole API
    wait, and its 4,807 requests and 1,674 replies of 2xx would fall to 159 and 52. Those 503s are
    faults to find, not requests to slow down.
  • The guess goes back to 1 s on any answer that does not ask to wait (maintainer). Going back
    only on a 2xx was built and measured: cwa-verification's createTeleTan answers 400 to invented
    values before it counts its quota, so with the rule chosen it gets 12 short waits and 62 refusals a
    minute, against 6 and 35. A 2xx depends on the invented values too, though, and an operation for
    which a valid request is rarely built would otherwise wait the longest guess after every 429.
  • The refused request is not sent again. Every turn builds a fresh request, so resending buys
    nothing a new turn does not. EvoMaster resends because it sends one request at a time.
  • The engine holds what it already has during a wait of the whole API. The plan's review found
    that otherwise up to sixteen queued requests would reach the API inside its quiet.
    • The code review then found that a hold ending at the deadline let them out after the run's time.
      They are now dropped when the run stops sending, and counted under noReply, which the docs say.
    • Requests already queued for a single operation when it is told to wait still go out; the known
      limitations say so.
  • Closing the engine gives back every request still waiting for a slot, not only held ones.
    Before, such a request went out after the run was over, or failed on its own thread and counted as
    lost.
  • The settings are in schedule.*, beside 13.1's stop, because the loop decides the wait.
    • The switch is listed among those that are not levers.
    • Because which turns are skipped depends on when the API asked, the file for getting the seed back
      turns it off, and the line under the seed speaks while it is on.
  • Small departures from the approved plan, all from the reviews:
    • Interaction.responseHeaderValues(name) rather than responseHeaders(), reusing the package's
      lookup without regard to case.
    • toldToWait.idle at the top of the block, since a round where every operation waits is the
      API's asking too.
    • One check of whether an operation may be sent, once there is room, rather than two.
  • Reviewer findings not acted on:
    • The idle time while the engine is held but the loop is waiting elsewhere (for an opening lap's
      step) is not counted as the API's. It is bounded by the lap's patience.
    • The command test's helpers are not shared with 13.1's command test.
    • The status code is computed twice per reply.
    • There is no loop test for a round where one operation waits and another's request could not be
      built: the generator always ends up building a request it can send, so no document produces that
      round on demand. The fix is two lines and is in the code.

Open questions

None.


  • Targets master
  • English throughout, including comments and test names
  • The hard rules in AGENTS.md hold
  • Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code

…API once two do

A reply of 429, or of 503 with Retry-After, makes the operation that drew it
wait for as long as the API says, or for a wait that doubles from a second to
a minute when it does not say; every other operation goes on. When a second
operation is told to wait while another waits, nothing at all is sent until
the whole API's wait is over, the requests the engine already holds included.
report.json says who was told to wait under toldToWait, and the summary how
much of the idle time the API asked for. One switch, schedule.waitWhenToldTo,
on by default, and two settings for the guessed wait. ADR-0042.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura ssegura changed the title Leave an operation alone when the API asks it to wait, and the whole API once two do Leave an operation or API alone when the API asks it to wait Oct 29, 2026
…indows too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura
ssegura merged commit cc3d907 into master Oct 29, 2026
11 checks passed
@ssegura
ssegura deleted the feat/m13-2a-wait-when-told-to-wait branch October 29, 2026 11:15
ssegura added a commit that referenced this pull request Oct 30, 2026
…asks, keep to a rate, and Maven Central (#431)

**Related:** None. This is the release of 2.4.0, with what `master`
holds on 30 October: #422 to #429.

## What you can do now that you could not before

- **Get 2.4.0.** Once this is merged and tagged, it installs four ways,
as 2.3.0 does:
  - `brew install isa-group/tap/restest`;
  - `docker run ghcr.io/isa-group/restest:2.4.0`;
  - the release's archives, with Java alone;
  - or built from the source.
- **What 2.4.0 adds:**
- a run whose opening lap the API answers with `401` throughout stops
there and ends with `5` (#422);
- an operation answered `429`, or `503` with `Retry-After`, is left
alone for as long as the API asks (#424);
- `--set engine.mostRequestsPerMinute=<n>` keeps a run to a rate, evenly
spaced (#425);
- the eight modules go to Maven Central as
`es.us.isa.restest:restest-*`, from this release on (#429);
- fixes: both reports kept when a quoted value is cut inside an emoji
(#427), `F920` and `F921` reported on operations RESTest read whole
(#428), and every Jackson library on a version with no known flaw
(#423).
- **What stays the same as in 2.3.0:** the command line, plans and
dictionaries. A run can end with one exit code more, `5`, and
`report.json` only gains members.

## Try it yourself

From a clean clone, with only Java and git:

```bash
git clone --quiet --branch release/2.4.0 https://github.com/isa-group/RESTest.git restest-2.4.0 && cd restest-2.4.0
./mvnw -q install -DskipTests
./restest version
./restest run restest-cli/src/test/resources/pet-shelter.yaml --url http://127.0.0.1:1 --budget 3s; echo "exit $?"
ls restest-out
grep -o '"verdict" *: *"[A-Za-z]*"' restest-out/report.json
dist/assemble.sh ../release --archives > /dev/null && ls ../release
tar -xzf ../release/restest-2.4.0.tgz -C ../release && ../release/restest-2.4.0/bin/restest version
```

Below is the real output, from a fresh clone of this branch on 30
October; the Java line will be your own. Nothing listens on port 1, so
the run ends with `3`, and it still writes both files:

```
RESTest 2.4.0
Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64
RESTest testing Pet Shelter at http://127.0.0.1:1

4 of 4 operations can be tested, seed -4339058821541680037, budget 3s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent

64 requests to 4 operations in 363ms, 90% of it idle
  opening lap: 4 requests in 46ms, 0 of 4 operations answered 2xx
  64 no reply
  1 of them were pushing at the API with values nobody sensible would send
  3 creation(s) meant to begin a series about a thing the run created were not accepted, so no series began
no faults found
report written to restest-out/report.json (9.6 KiB)
page written to restest-out/report.html (90.7 KiB)
the run itself was not kept; pass --store to keep every request and reply
restest: nothing at http://127.0.0.1:1 answered any of the 64 requests, so the API was never actually tested. Check the address, and that it is running.
exit 3
report.html
report.json
"verdict":"nothingAnswered"
restest-2.4.0
restest-2.4.0.tgz
restest-2.4.0.zip
RESTest 2.4.0
Java 25.0.4.1 (Eclipse Adoptium), Mac OS X 27.0.1 aarch64
```

With Docker as well, the image is built from the same clone, as the
release builds it:

```bash
docker build --quiet --tag restest:2.4.0 . > /dev/null && docker run --rm restest:2.4.0 version
```

```
RESTest 2.4.0
Java 21.0.13 (Eclipse Adoptium), Linux 7.0.14-linuxkit aarch64
```

What the tag will leave on the Central Portal, laid out from the same
clone, now with no `-SNAPSHOT` in any name:

```bash
./mvnw -q package -Pcentral -DskipTests -pl restest-cli -am && dist/central.sh target/central
ls target/central/maven-central/es/us/isa/restest
ls target/central/maven-central/es/us/isa/restest/restest-core/2.4.0
grep -c "<dependency>" target/central/maven-central/es/us/isa/restest/restest-cli/2.4.0/restest-cli-2.4.0.pom
ls restest-cli/target/lib | wc -l
```

```
/…/restest-2.4.0/target/central/maven-central
restest-cli
restest-core
restest-exec
restest-gen
restest-oracles
restest-report
restest-spec
restest-store
restest-core-2.4.0-javadoc.jar
restest-core-2.4.0-javadoc.jar.md5
restest-core-2.4.0-javadoc.jar.sha1
restest-core-2.4.0-sources.jar
restest-core-2.4.0-sources.jar.md5
restest-core-2.4.0-sources.jar.sha1
restest-core-2.4.0.jar
restest-core-2.4.0.jar.md5
restest-core-2.4.0.jar.sha1
restest-core-2.4.0.pom
restest-core-2.4.0.pom.md5
restest-core-2.4.0.pom.sha1
70
      70
```

## How it works

**The version.** `-SNAPSHOT` is how Maven writes "on the way to this
version", and this pull request removes it from every `pom.xml`. After
the merge, the tag `v2.4.0` on the merge commit starts the release
workflow ([`docs/ci.md`](docs/ci.md#releases)):
- it builds the image and the archives, and checks them on three
operating systems and two processors;
- it installs the Homebrew formula for the archive on macOS and Linux;
- it signs the eight modules and uploads them to the Central Portal,
once a maintainer approves the `central` job; the Portal validates them
and publishes nothing;
- it pushes the image, and leaves a draft release for a maintainer to
write the notes in and publish.

A maintainer publishes the deployment on the Portal before publishing
the release, and publishing the release starts the Homebrew workflow,
which writes 2.4.0 into the tap.

**The base images.** Step 1 of every release moves the `Dockerfile`'s
two base images to the newest patch published today. This time the Java
in the image moves too, from 21.0.12.1 to 21.0.13, as the output above
shows, and the pages that print it say so.

**The pages.** The README and chapters 2 and 9 of the manual name the
version to install, and now name 2.4.0. Every other page was written for
2.4.0 as each change landed, from #422 to #429.

## What changed in the code

No Java code changed, and nothing under `src/`.

- **Version:** every `pom.xml`, from `2.4.0-SNAPSHOT` to `2.4.0`, ten
files. The JSON Schemas' `$id` already names `v2.4.0` since #421.
- **`Dockerfile`:** the digests of `eclipse-temurin:25-jdk` and
`eclipse-temurin:21-jre` as they are today.
- **`README.md`:** the installation section fetches, clones and runs
`2.4.0`. Its `restest version` output is the image's above.
- **`docs/manual/02-installing.md`:** the same for Docker, the archive,
the checksums, the source and Updating. The Docker and archive outputs
are those above. The two lines under Homebrew's `restest version` are
what this pull request's run of `release.yml` printed in
`check-homebrew` on macOS, as `docs/ci.md` asks:

  ```
  RESTest 2.4.0
  Java 21.0.12.1 (Homebrew), Mac OS X 26.6.2 aarch64
  ```

They are the lines chapter 2 already had once `2.3.0` became `2.4.0`,
since the formula still installs Homebrew's `openjdk@21` 21.0.12.1 on
macOS; only the version changed.
- **`docs/manual/09-exit-codes-and-ci.md`:** the workflow fetches and
runs the 2.4.0 archive. Its sentence on releases before 2.3.0, which
write no page, stays as it is.
- **`docs/DESIGN.md`:** the stack table named `jackson-core` 2.22.1,
which #423 moved to 2.22.3 in the root POM; it now says 2.22.3.

## Evidence

- [x] **Tests:** `./mvnw verify` is green at `d19edf0f`, with version
2.4.0: 4,945 tests, 0 failures, BUILD SUCCESS in 2 min 33 s. 141 of them
are the architecture and documentation rules, `PublishedSchemasTest`
among them, which on a release requires the schemas' `$id` to name it.
CI on `5baafe06`:
[CI](https://github.com/isa-group/RESTest/actions/runs/41073186888) and
[Release](https://github.com/isa-group/RESTest/actions/runs/41073186769),
19 checks green, and `central` and `publish` skipped, as on every pull
request. In that Release run, `check-homebrew` installed the formula for
this branch's archive on macOS and on Ubuntu: 625 MB and 823 MB, as the
pages say. *Java 25 on windows-latest* failed on its first attempt in
`RunLoopTest.a_slow_401_to_the_lap_stops_the_run_when_it_arrives`, which
took 4.16 s where it allows less than 4 s; the same row passed when
re-run, and that test passed on every row of `master`'s last runs.
Nothing under `src/` changed here.
- [x] **Architecture tests:** unchanged, none weakened.
- [x] **Mutation score:** no oracle or constraint touched.
- [x] **Per-request overhead:** unchanged; nothing under `src/` changed.
- [x] **Smoke run:** unchanged, for the same reason; CI runs it on this
pull request.
- [x] **Idle time:** unchanged.
- [x] **Artefact:** `check-restest.sh 2.4.0` passes on both the archive
and the image built here, every check `ok`.
  - **The archive**, `--launcher` on `restest-2.4.0/bin/restest`:
    - version;
    - `JAVA_TOOL_OPTIONS`;
    - an unknown option answering `2`;
    - a run with no answer ending with `3` and its files;
    - the schemas and `THIRD-PARTY.txt` in the folder;
    - `127` with no Java and with a `JAVA_HOME` that holds none;
    - a symbolic link.
- **The image**, `--image restest:2.4.0`: the same, and `docker stop`
ending a run with `143` and a report that says it was cut short.
- **What Maven Central gets:** `dist/central.sh` passes on the build
above, with every check it makes, fetching each module offline with
Maven among them; `restest-cli`'s POM names the 70 libraries
`restest-cli/target/lib` holds, which the archive's `lib/` carries
beside `restest-cli` itself.

## Decisions taken

- **No release evaluation on the Web Fuzzing Dataset, as for 2.3.0.**
The three safeguards change a run only against an API that answers `401`
to the whole opening lap (#422), that asks to wait (#424), or when a
rate is named (#425, off otherwise); #428 was measured across the
dataset's 52 documents in its own pull request; #427 and #423 change no
request.
- **The base images are today's patch,** as step 1 of a release asks
while Dependabot is off. The image's Java moves from 21.0.12.1 to
21.0.13, so the `restest version` lines of the README and chapter 2 that
come from the image move with it.
- **Chapter 2's Homebrew lines come from this pull request's CI,** not
from a Mac here. 2.4.0 cannot be installed from the tap before it is
published, and `docs/ci.md` names `check-homebrew`'s output as their
source. They needed no commit of their own: what the job printed is what
the page already said.
- **`docs/DESIGN.md`'s `jackson-core` version is corrected here,** being
one line in the release's own stack table; the rest of the table agrees
with the root POM, the wrapper and the `Dockerfile`.
- **The release notes are not in this pull request.** They go into the
draft release after the tag, as `docs/ci.md` says.

## Open questions

- **A new Dependabot alert,
[48](https://github.com/isa-group/RESTest/security/dependabot/48), high,
opened today:** Guava up to 33.7.1 allocates memory without bound when
it deserialises some of its collections, fixed in 33.7.2. The release
carries `guava-33.5.0-jre.jar`, which the OpenAPI parser and the HTTP
client bring and the root POM pins. RESTest's own code uses neither
Guava nor Java serialisation. Whether 2.4.0 waits for a pull request
that moves Guava, as #423 did for Jackson, is the maintainer's call;
this one does not touch it.
- **After the merge, each step taken by a maintainer:**
- the tag `v2.4.0` on the merge commit, and the approval of the
`central` job when the run asks for it;
  - the notes in the draft;
- the deployment `RESTest 2.4.0 run <run>` published on the Central
Portal, and then the release, which also starts the Homebrew workflow
for 2.4.0;
- the README's installation commands tried from outside, `brew install
isa-group/tap/restest` among them, and
`https://repo1.maven.org/maven2/es/us/isa/restest/restest-cli/2.4.0/`
answering;
- the pull request that moves `master` to `2.5.0-SNAPSHOT`, and the
schemas' `$id` to `v2.5.0`.

---

- [x] Targets `master`
- [x] English throughout, including comments and test names
- [x] The hard rules in `AGENTS.md` hold
- [x] Nothing from "Out of scope" in `docs/DESIGN.md` without explicit
approval

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant